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Abstract. Scalability issues may prevent users from verifying critical proper¬ 
ties of a complex hardware design. In this situation, we propose to synthesize a 
“safety shield” that is attached to the design to enforce the properties at run time. 
Shield synthesis can succeed where model checking and reactive synthesis fail, 
because it only considers a small set of critical properties, as opposed to the com¬ 
plex design, or the complete specification in the case of reactive synthesis. The 
shield continuously monitors the input/output of the design and corrects its erro¬ 
neous output only if necessary, and as little as possible, so other non-critical prop¬ 
erties are likely to be retained. Although runtime enforcement has been studied 
in other domains such as action systems, reactive systems pose unique challenges 
where the shield must act without delay. We thus present the first shield synthesis 
solution for reactive hardware systems and report our experimental results. This 
is an extended version of (5 ], featuring an additional appendix. 


1 Introduction 


Model checking 1110118 1 can formally verify that a design satisfies a temporal logic 
specification. Yet, due to scalability problems, it may be infeasible to prove all critical 
properties of a complex design. Reactive synthesis «T7l4l is even more ambitious since 
it aims to generate a provably correct design from a given specification. In addition 
to scalability problems, reactive synthesis has the drawback of requiring a complete 
specification, which describes every aspect of the desired design. However, writing a 
complete specification can sometimes be as hard as implementing the design itself. 

We propose shield synthesis as a way 
to complement model checking and reactive 
synthesis. Our goal is to enforce a small set 
of critical properties at runtime even if these 
properties may occasionally be violated by 
the design. Imagine a complex design and a 
set of properties that cannot be proved due to 
scalability issues or other reasons (e.g., third-party IP cores). In this setting, we are 
in good faith that the properties hold but we need to have certainty. We would like to 
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Fig. 1: Attaching a safety shield. 
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automatically construct a component, called the shield, and attach it to the design as 
illustrated in Fig. [T] The shield monitors the input/output of the design and corrects the 
erroneous output values instantaneously, but only if necessary and as little as possible. 

The shield ensures both correctness and minimum interference. By correctness, we 
mean that the properties must be satisfied by the combined system, even if they are oc¬ 
casionally violated by the design. By minimum interference, we mean that the output 
of the shield deviates from the output of the design only if necessary, and the deviation 
is kept minimum. The latter requirement is important because we want the design to 
retain other (non-critical) behaviors that are not captured by the given set of properties. 
We argue that shield synthesis can succeed even if model checking and reactive syn¬ 
thesis fail due to scalability issues, because it has to enforce only a small set of critical 
properties, regardless of the implementation details of a complex design. 

This paper makes two contributions. First, we define a general framework for solv¬ 
ing the shield synthesis problem for reactive hardware systems. Second, we propose 
a new synthesis method, which automatically constructs a shield from a set of safety 
properties. To minimize deviations of the shield from the original design, we propose a 
new notion called k-stabilization: When the design arrives at a state where a property 
violation becomes unavoidable for some possible future inputs, the shield is allowed to 
deviate for at most k consecutive steps. If a second violation happens during the fc-step 
recovery phase, the shield enters a fail-safe mode where it only enforces correctness, 
but no longer minimizes the deviation. We show that the fc-stabilizing shield synthesis 
problem can be reduced to safety games HSJ. Following this approach, we present a 
proof-of-concept implementation and give the first experimental results. 

Our work on shield synthesis can complement model checking by enforcing any 
property that cannot be formally proved on a complex design. There can be more appli¬ 
cations. For example, we may not trust third-party IP components in our system, but in 
this case, model checking cannot be used because we do not have the source code. Nev¬ 
ertheless, a shield can enforce critical interface assumptions of these IP components at 
run time. Shields may also be used to simplify certification. Instead of certifying a com¬ 
plex design against critical requirements, we can synthesize a shield to enforce them, 
regardless of the behavior of the design. Then, we only need to certify this shield, or 
the synthesis procedure, against the critical requirements. Finally, shield synthesis is a 
promising new direction for synthesis in general, because it has the strengths of reactive 
synthesis while avoiding its weaknesses — the set of critical properties can be small and 
relatively easy to specify — which implies scalability and usability. 

Related work. Shield synthesis is different from recent works on reactive synthe¬ 
sis 117141121 , which revisited Church’s problem 119181191 on constructing correct sys¬ 
tems from logical specifications. Although there are some works on runtime enforce¬ 
ment of properties in other domains II20I14I13I . they are based on assumptions that do 
not work for reactive hardware systems. Specifically, Schneider |j20l proposed a method 
that simply halts a program in case of a violation. Ligatti et al. M used edit automata 
to suppress or insert actions, and Falcone et al. oca proposed to buffer actions and dump 
them once the execution is shown to be safe. None of these approaches is appropriate 
for reactive systems where the shield must act upon erroneous outputs on-the-fly, i.e., 
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without delay and without knowing what future inputs/outputs are. In particular, our 
shield cannot insert or delete time steps, and cannot halt in the case of a violation. 

Methodologically, our new synthesis algorithm builds upon the existing work on 
synthesis of robust systems 0, which aims to generate a complete design that satisfies 
as many properties of a specification as possible if assumptions are violated. However, 
our goal is to synthesize a shield component S, which can be attached to any design 
D , to ensure that the combined system (S o D) satisfies a given set of critical proper¬ 
ties. Our method aims at minimizing the ratio between shield deviations and property 
violations by the design, but achieves it by solving pure safety games. Furthermore, the 
synthesis method in 0 uses heuristics and user input to decide from which state to 
continue monitoring the environmental behavior, whereas we use a subset construction 
to capture all possibilities to avoid unjust verdicts by the shield. We use the notion of 
fc-stabilization to quantify the shield deviation from the design, which has similarities 
to Ehlers and Topcu’s notion of fc-resilience in robust synthesis lfT2l forGR(l) specifi¬ 
cations 0 . However, the context of our work is different, and our /.'-stabilization limits 
the length of the recovery period instead of tolerating bursts of up to k glitches. 
Outline. The remainder of this paper is organized as follows. We illustrate the technical 
challenges and our solutions in Section[2]using an example. Then, we establish notation 
in Section [3] We formalize the problem in a general framework for shield synthesis in 
Section[4] and present our new method in Section[5] We present our experimental results 
in Section[6]and, finally, give our conclusions in Section[7] 

2 Motivation 

In this section, we illustrate the challenges associated with shield synthesis and then 
briefly explain our solution using an example. We start with a traffic light controller 
that handles a single crossing between a highway and a farm road. There are red (r) or 
green (g) lights for both roads. An input signal, denoted p € {0,1}, indicates whether 
an emergency vehicle is approaching. The controller takes p as input and returns h,f as 
output. Here, h £ { r. g] and f £ {r, g} are the lights for highway and farm road, respec¬ 
tively. Although the traffic light controller interface is simple, the actual implementation 
can be complex. For example, the controller may have to be synchronized with other 
traffic lights, and it can have input sensors for cars, buttons for pedestrians, and sophis¬ 
ticated algorithms to optimize traffic throughput and latency based on all sensors, the 
time of the day, and even the weather. As a result, the actual design may become too 
complex to be formally verified. Nevertheless, we want to ensure that a handful of safety 
critical properties are satisfied with certainty. Below are three example properties: 

1. The output gg — meaning that both roads have green lights — is never allowed. 

2. If an emergency vehicle is approaching ( p = 1), the output must be IT. 

3. The output cannot change from gr to rg, or vice versa, without passing IT. 

We want to synthesize a safety shield that can be attached to any implementation of this 
traffic light controller, to enforce these properties at run time. 

In a first exercise, we only consider enforcing Properties 1 and 2. These are simple 
invariance properties without any temporal aspects. Such properties can be represented 
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by a truth table as shown in Fig. [2] Cleft). We use 0 to encode r, and 1 to encode g. 

Forbidden behavior is marked in bold red. The shield must ensure both correctness 

and minimum interference. That is, it should only change the output for red entries. 

In particular, it should not ignore the 

design and hard-wire the output to IT. 

When p = 1 but the output is not rr, 

the shield must correct the output to rr. 

When p = 0 but the output is gg, the 

shield must turn the original output gg 

into either rg, gr, or rr. Assume that 

gr is chosen. As illustrated in Fig. [2] o ocf n ~ 

a & i—i Fig. 2: Enforcing Properties 1 and 2. 

(right), we can construct the transition 

functions h' = -ip A h and f' = -*p/\-*hf\ /, as well as the shield circuit accordingly. 

Next, we consider enforcing Properties 1-3 together. Property 3 brings in a temporal 
aspect, so a simple truth table does not suffice any more. Instead, we express the prop¬ 
erties by an automaton, which is shown in Fig. [3] Edges are labeled by values of phf, 
where p £ {0,1} is the controller’s input and h, f are outputs for highway and farm road. 
There are three non-error states: H denotes 
the state where highway has the green light, 

F denotes the state where farm road has the 
green light, and B denotes the state where 
both have red lights. There is also an error 
state, which is not shown. Missing edges lead 
to this error state, denoting forbidden situations, e.g., 1 gr is not allowed in state H. Al¬ 
though the automaton still is not a complete specification, the corresponding shield can 
prevent catastrophic failures. By automatically generating a small shield as shown in 
Fig. [II our approach has the advantage of combining the functionality and performance 
of the aggressively optimized implementation with guaranteed safety. 



Fig. 3: Traffic light specification. 


While the shield for Property 1 and 2 could be realized by purely combinational 
logic, this is not possible for the specification in Fig. [3] The reason is the temporal 
aspect brought in by Property 3. For example, if we are in state F and observe Ogg, 
which is not allowed, the shield has to make a correction in the output signals to avoid 
the violation. There are two options: changing the output from gg to either rg or rr. 
However, this fix may result in the next state being either B or F. The question is, 
without knowing what the future inputs/outputs are, how do we decide from which state 
the shield should continue to monitor the behavior of the design in order to best detect 
and correct future violations? If the shield makes a wrong guess now, it may lead to a 
suboptimal implementation that causes unnecessarily large deviation in the future. 


To solve this problem, we adopt the most conservative approach. That is, we assume 
that the design V meant to give one of the allowed outputs, so either rr or rg. Thus, our 
shield continues to monitor the design from both F and B. Technically, this is achieved 
by a form of subset construction (see Sec. l5.2l >. which tracks all possibilities for now, 
and then gradually refines its knowledge with future observations. For example, if the 
next observation is Ogr, we assume that the design V meant rr earlier, and so it must be 
in B and traverse to H. If it were in F, we could only have explained Ogr by assuming 
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a second violation, which is less optimistic than we would like to be. In this work, we 
assume that a second violation occurs only if an observation is inconsistent with all 
states that it could possibly be in. For example, if the next observation is not Ogr but 
1 rg, which is neither allowed in F nor in B, we know that a second violation occurs. Yet, 
after observing 1 rg, we can be sure that we have reached the state B, because starting 
from both F and B, with input p = 1, the only allowed output is rr, and the next state is 
always B. In this sense, our construction implements an “innocent until proved guilty” 
philosophy, which is key to satisfy the minimum interference requirement. 

To bound the deviation of the shield when a property violation becomes unavoid¬ 
able, we require the shield to deviate for at most k consecutive steps after the initial 
violation. We shall formalize this notion of k-stabilization in subsequent sections and 
present our synthesis algorithm. For the safety specification in Fig. [3 our method would 
reduce the shield synthesis problem into a set of safety games , which are then solved 
using standard techniques (cf. 1131 ). We shall present the synthesis results in Section[ 6 ] 


3 Preliminaries 

We denote the Boolean domain by B = {true, false}, denote the set of natural numbers 
by N, and abbreviate N U {oo} by N°°. We consider a reactive system with a finite 
set I = {*i,..., i m } of Boolean inputs and a finite set O = {oi,..., o n } of Boolean 
outputs. The input alphabet is 17/ = 2 1 , the output alphabet is So = 2°, and S = 
Si x So - The set of finite (infinite) words over S is denoted by S* (17“), and 17*’“ = 
S* U 17“. We will also refer to words as (execution) traces. We write |cf| for the length 
of a trace a G 17*’“. For WJ = XoXi ... G Sf and Wo = yoV\ • ■ ■ G Sq, we write 
Wj\\Wo for the composition (a;o, yo){x\, yi )... G 17 “. A set L C 17“ of infinite words 
is called a language. We denote the set of all languages as C = 2 s . 

Reactive Systems. A reactive system T> = (Q. qo , 17/, So, 8, A) is a Mealy machine, 
where Q is a finite set of states, qo G Q is the initial state, <5 : Q x Si —>• Q is a 
complete transition function, and A : Q x Si —> So is a complete output function. 
Given the input trace Wj = xqx \... G Sf, the system V produces the output trace 
Wo = S>(WJ) = X(qo,xo)X(qi,Xi )... G Sq, where qt + i = 5(qi,Xi ) for all i > 0. The 
set of words produced by T> is denoted L(T>) = {Wj\\Wo G 17“ | V(WJ) = <ro}- We 
also refer to a reactive system V as a (hardware) design. 

Let V = (Q,qo, Si, So,8, X) and V = (Q 1 ,q' 0 , S, So,8', A') be reactive sys¬ 
tems. Their serial composition is constructed by feeding the input and output of V to 

V as input. We use V o V' to denote such a composition ( Q, qo, Si, So, 8, A), where 
Q = Q X Q', q 0 = ( q 0 ,q' 0 ), K(q,q'),ai) = (8(q, 07 ), 8'(q', (cr/, X(q, 07 )))), and 
A((< 7 , q'), <Ji) = X'(q', ( 01 , X(q, ct/))). 

Specifications. A specification ip defines a set L(ip) C 17“ of allowed traces. A spec¬ 
ification (p is realizable if there exists a design V that realizes it. T) realizes p, written 

V \= ip, iff L(V) C L(ip). We assume that ip is a (potentially incomplete) set of 
properties {pi, ■ ■ ■, pi} such that L(p ) = f), L(ipi), and a design satisfies ip iff it sat¬ 
isfies all its properties. In this work, we are concerned with a safety specification p s , 
which is represented by an automaton ip s = ( Q, qo, S, 8, F ), where S = Si U So, 
8 : Q x 17 —»• Q, and F C Q is a set of safe states. The run induced by trace 
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a = a o<j i ... £ S u is the state sequence q = qoqi... such that qt + 1 = 5{qi,<Ji). 
Trace a (of a design V) satisfies ip s if the induced run visits only the safe states, i.e., 
Vi > 0 . qt £ F. The language L{tp s ) is the set of all traces satisfying ip s . 

Games. A (2-player, alternating) game is a tuple Q = (G, go, Si, So, <5, win), where 
G is a finite set of game states, go £ G is the initial state, S : G x Si x So —»• G is 
a complete transition function, and win : G“ —» B is a winning condition. The game is 
played by two players: the system and the environment. In every state g £ G (starting 
with go), the environment first chooses an input letter 07 £ Si, and then the system 
chooses some output letter 00 £ So- This defines the next state g' = S{g, <ji, cro), and 
so on. The resulting (infinite) sequence g = go<j\ ... of game states is called a play. A 
play is won by the system iff win)#) is true. 

A safety game defines win via a set F 9 C G of safe states: win(#o5i • • ■) is true iff 
Vi > 0 - Pi £ F 9 , i.e., if only safe states are visited. A (memoryless) strategy for the 
system is a function p : G x Si —> So- A strategy is winning for the system if all plays 
g that can be constructed when defining the outputs using the strategy satisfy win(#). 
The winning region is the set of states from which a winning strategy exists. We will 
use safety games to synthesize a shield, which implements the winning strategy in a 
new reactive system S = (G, qo,Si, So, S’, p) with S'(g, of) = 5(g, oi,p(g, cti)). 


4 The Shield Synthesis Framework 

We define a general framework for shield synthesis in this section before presenting a 
concrete realization of this framework in the next section. 

Definition 1 (Shield). Let V = (Q, qo, Si, So, 5, A) be a design, ip be a set of prop¬ 
erties, and ip v C p be a valid subset such that T> \= ip v . A reactive system S = 
(■ Qq'o, S, So, S', A') is a shield ofV with respect to (<p \ p v ) iff (V o S) |= ip. 

Here, the design is known to satisfy <p v C <p. Furthermore, we are in good faith that T> 
also satisfies ip\ip v , but it is not guaranteed. We synthesize S, which reads the input 
and output of V while correcting its erroneous output as illustrated in Fig.Q] 

Definition 2 (Generic Shield). Given a set ip = <p v U (<p \ ip v ) of properties. A reactive 
system S is a generic shield iff it is a shield of any design T> such that T> |= <p v . 

A generic shield must work for any design V \= <p v . Hence, the shield synthesis proce¬ 
dure does not need to consider the design implementation. This is a realistic assumption 
in many applications, e.g., when the design 'D comes from the third party. Synthesis of a 
generic shield also has a scalability advantage since the design V, even if available, can 
be too complex to analyze, whereas ip often contains only a small set of critical proper¬ 
ties. Finally, a generic shield is more robust against design changes, making it attractive 
for safety certification. In this work, we focus on the synthesis of generic shields. 

Although the shield is defined with respect to ip (more specifically, ip\ip v ), we must 
refrain from ignoring the design completely while feeding the output with a replacement 
circuit. This is not desirable because the original design may satisfy additional (non- 
critical) properties that are not specified in p but should be retained as much as possible. 
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In general, we want the shield to deviate from the design only if necessary, and as little 
as possible. For example, if V does not violate ip, the shield S should keep the output 
of D intact. This rationale is captured by our next definitions. 

Definition 3 (Output Trace Distance Function). An output trace distance function 
(OTDF) is a function d a : Eq U x Eq U —> N°° such that 

1. d a {Woi of)') = 0 when Wo = ~oo ; 

2. d a (Wooo,of)' oo) = d a (oo,Wo') when 00 = 00 , and 

3. d a (oooo,oo'oo') > d a ( 00 , 00 ') when o 0 ± 00 '■ 

An OTDF measures the difference between two output sequences (of the design D 
and the shield S ). The definition requires monotonicity with respect to prefixes: when 
comparing trace prefixes with increasing length, the distance can only become larger. 

Definition 4 (Language Distance Function). A language distance function (LDF) is 
a function d L : C x E u —> N°° such that \/L g C,o g E u .0 g L —y d L (L , o) = 0. 

An LDF measures the severity of specification violations by the design by mapping a 
language (of p) and a trace (of T>) to a number. Given a trace 7r g E !jJ , its distance to 
L(p) is 0 if cf satisfies p. Greater distances indicate more severe specification violations. 
An OTDF can (but does not have to) be defined via an LDF by taking the minimum 
output distance between 0 = (Wj\ \Wo) and any trace in the language L: 

T f— 1 !™ d a (Wo',Wo) if 3oo'& ■ (wj\\Wo') & L 

d L (L, oj\\oo) = < vdWo'eL 

I 0 otherwise. 

The input trace is ignored in d a because the design V can only influence the output. If 
no alternative output trace makes the word part of the language, the distance is set to 0 to 
express that it cannot be the design’s fault. If L is defined by a realizable specification 
p, this cannot happen anyway, since VoJ g Ef . 3Wo € Eq .(Wj\\Wo) g L(p) is a 
necessary condition for the realizability of p. 

Definition 5 (Optimal Generic Shield). Let p be a specification, p v C p be the valid 
subset, d a be an OTDF, and d L be an LDF. A reactive system S is an optimal generic 
shield if and only if for all WJ g Ef and Wo € Eq, 

(oJ\\oo)£L(t v ) L(p),Wj\\S(oj\\od) ) = 0 A (1) 

d a (W5,S(oj\\o5)) < d L (L(p),Wj\\Wo)) ■ (2) 

The implication means that we only consider traces that satisfy p v since D |= p" is 
assumed. This can be exploited by synthesis algorithms to find a more succinct shield. 
Part (jTJ of the implied formula ensures correctness: Do S must satisfy Part (0 
ensures minimum interference: “small” violations result in “small” deviations. Def.[5]is 
designed to be flexible: Different notions of minimum interference can be realized with 
appropriate definitions of d rr and d L . One realization will be presented in Section [K] 

1 Applying d L instead of “C L(p)” adds flexibility: the user can define d L in such a way that 
d L (L,W ) = 0 even if a £ L to allow such traces as well. 
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Proposition 1. An optimal generic shield S cannot deviate from the design’s output 
before a specification violation by the design V is unavoidable. 

Proof If there has been a deviation d a (Wo , S(aj\\ao)) 7 ^ 0 on the finite input prefix 
< 7 , but this prefix can be extended into an infinite trace o' such that d L (L(ip),o’) = 0, 
meaning that a violation is avoidable, then Part (0 of Def.[5]is violated because of the 
(prefix-)monotonicity of d a (the deviation can only increase when the trace is extended), 
and the fact that d a < d L is false if d a f 0. □ 

5 Our Shield Synthesis Method 

In this section, we present a concrete realization of the shield synthesis framework by 
defining OTDF and LDF in a practical way. We call the resulting shield a k-stabilizing 
generic shield. While our framework works for arbitrary specifications, our realization 
assumes safety specifications. 

5.1 fc-Stabilizing Generic Shields 

A /.-stabilizing generic shield is an optimal generic shield according to Def.[5] together 
with the following restrictions. When a property violation by the design V becomes un¬ 
avoidable (in the worst case over future inputs), the shield S is allowed to deviate from 
the design’s outputs for at most k consecutive time steps, including the current step. 
Only after these k steps, the next violation is tolerated. This is based on the assumption 
that specification violations are rare events. If a second violation happens within the 
fc-step recovery period, the shield enters a fail-safe mode, where it enforces the critical 
properties, but stops minimizing the deviations. More formally, a /.'-stabilizing generic 
shield requires the following configuration of the OTDF and LDF functions: 

1. The LDF d L (L(ip),o) is defined as follows: Given a trace <7 £ S u , its distance to 
L(tp) is 0 initially, and increased to 00 when the shield enters the fail-safe mode. 

2. The OTDF function d a (of), of)') returns 0 initially, and is set to 00 if ooi 7 ^ cro'i 
outside of a fc-step recovery period. 

To indicate whether the shield is in the fail-safe mode or a recovery period, we add a 
counter c £ {0,..., fc}. Initially, c is 0. Whenever there is a property violation by the 
design, c is set to fc in the next step. In each of the subsequent steps, c decrements until it 
reaches 0 again. The shield can deviate if the next state has c > 0. If a second violation 
happens when c > 1, then the shield enters the fail-safe mode. A 1-stabilizing shield 
can only deviate in the time step of the violation, and can never enter the fail-safe mode. 

5.2 Synthesizing fc-Stabilizing Generic Shields 

The flow of our synthesis procedure is illustrated in Fig. [4] Let <p = . .. ,ipi} be 

the critical safety specification, where each pi is represented as an automaton = 
{Qiido.i: £■> Si, F%)- The synchronous product of these automata is again a safety au¬ 
tomaton. We use three product automata: Q = {(). r/o, S, 6. F) is the product of all 
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Fig. 4: Outline of our fc-stabilizing generic shield synthesis procedure. 



Fig. 5: The safety automaton 7Z. 


Fig. 6: The deviation monitor T. 


properties in ip; V = (V, vq, S, S v , F v ) is the product of properties in <p v C ip\ and 
1Z = (R, 7"o, S, S r ,F r ) is the product of properties in ip \ ip v . Starting from these au¬ 
tomata, our shield synthesis procedure consists of five steps. 

Step 1. Constructing the Violation Monitor U\ From 1Z , which represents <p \ ip v , we 
build U = ([/, Mo, S, 5 U ) to monitor property violations by the design. The goal is to 
identify the latest point in time from which a specification violation can still be corrected 
with a deviation by the shield. This constitutes the start of the recovery period. 

The first phase of this construction (Step 1-a) is to consider the automaton 7 Z = 
(7?, 7 - 0 , F, S r , F r ) as a safety game and compute its winning region W r C F r . The 
meaning of W r is such that every reactive system V |= (pp \ p> v ) must produce outputs 
in such a way that the next state of 7Z stays in W r . Only when the next state of 1Z would 
be outside of W r , our shield will be allowed to interfere. 

Example 1. Consider the safety automaton 7Z in Fig. [3 where i is an input, o is an 
output, and r x is unsafe. The winning region is W = {ro} because from r\ the input i 
controls whether r x is visited. The shield must be allowed to deviate from the original 
transition ?’o —»• r\ if o ^ i. In r i it is too late because visiting an unsafe state cannot 
be avoided any more, given that the shield can modify the value of o but not i. □ 

The second phase (Step 1-b) is to expand the state space from It to 2 R via a subset 
construction. The rationale behind it is as follows. If the design makes a mistake (i.e., 
picks outputs such that 7 Z enters a state r W r from which the specification cannot 
be enforced), we have to “guess” what the design actually meant to do in order to find 
a state from which we can continue monitoring its behavior. We follow a generous 
approach in order not to treat the design unfairly: we consider all output letters that 
would have avoided falling out of W r , and continue monitoring the design behavior 
from all the corresponding successor states in parallel. Thus, U is essentially a subset 
construction of 1Z , where a state u&UofU represents a set of states in 1Z. 
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The third phase (Step 1-c) is to expand the state space of U by adding a counter c £ 
{0,..., k} as described in the previous subsection, and adding a special fail-safe state 
ue■ The final violation monitor is U = ( U, uq, E, S u ), where U = ( 2 R x {0,..., A:}) U 
ue is the set of states, uq = ({fo}, 0) is the initial state, E is the set of input letters, 
and 5 U is the next-state function, which obeys the following rules: 

1. 5 u (ue 1 a) = ue (meaning that ue is a trap state), 

2. 5 u ((u , c), a) = ue if c > 1 and Vr £ u : 5 r (r, a) ^ W r , 

3. 5 u ((u,c), (a i, (To)) = ({r'elT | 3r e u,oo £ E 0 ■6 r (r, (cr/,cro / )) = r'},k) 
if c < 1 and Vr£u. S r (r, ( 07 , ao )) ^ W r , and 

4. S u ((u, c),a) = ({r' £W r \3r£u.S r (r, a) = r'},dec(c)) if 3r £u. 6 r (r, a) £W r , 
where dec(0) = 0 and dec(c) = c — 1 if c > 0. 

Our construction sets c = k whenever the design leaves the winning region, and not 
when it enters an unsafe state. Hence, the shield S can take remedial action as soon as 
the “the crime is committed”, before the damage is detected, which would have been 
too late to correct the erroneous outputs of the design. 

Example 2. We illustrate the construction of U using the specification from Fig. 0 
which is a safety automaton if we make all miss¬ 
ing edges point to an (additional) unsafe state. 

The winning region consists of all safe states, 
i.e., W r = {H,B,F}. The resulting viola¬ 
tion monitor is U = ({H, B, F, HB, FB, HFB} x 
{0,..., k} U ue, (H, 0), E, S u ), where S u is il¬ 
lustrated in Fig.|7]as a table (the graph would be 
messy), which lists the next state for all possible 
present states as well as inputs and outputs by 
the design. Lightning bolts denote specification 
violations. The update of the counter c, which is not included in Fig. [7] is as follows: 
whenever the design commits a violation (indicated by lightning) and c < 1, then c is 
set to k. If c > 1 at the violation, the next state is 11 p;■ Otherwise, c is decremented. □ 

Step 2. Constructing the Validity Monitor V': From V = (V, vu. E, 5 V ,F V ), which 
represents tp v , we build an automaton V' to monitor the validity of ip v by solving a 
safety game on V and computing the winning region W v C F v . We will use W v 
to increase the freedom for the shield: since we assume that V |= ip v , we are only 
interested in the cases where V never leaves W v . If it does, our shield is allowed to 
behave arbitrarily from that point on. We extend the state space from V to V' by adding 
a bit to memorize if we have left the winning region W v . Hence, the validity monitor 
is defined as V' = ( V', Vq, E, S v> , F v '), where V' = B x V is the set of states, Vq = 
{false, no} is the initial state, S v '((b, v),a) = (b', S v (v, a)), where b' = true if b = true 
or S v (v, a) qL W v , and b 1 = false otherwise, and F v ' = {(&, v) £ V' \ b = false}. 

Step 3. Constructing the Deviation Monitor T: We build T = (T, t 0 , Eo x Eq. S f ) to 
monitor the deviation of the shield’s output from the design’s output. Here, T = {to, f 1 } 
and (5*(t, (ao, cro')) = to iff ao = cc/. That is, T will be in t\ if there was a deviation 
in the last time step, and in to otherwise. This deviation monitor is shown in Fig. [6] 
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Fig. 7: 6" for the spec from Fig. [3] 
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Step 4. Constructing the Safety Game Q\ Given the monitors U, V', T and the au¬ 
tomaton Q, which represents (p, we construct a safety game Q = (G, go, Ej x So, Eo, 
S 9 , F 9 ), which is the synchronous product of U , T, V' and Q, such that G = U x T x 
V 7 x Q is the state space, go = (u o, to,v' 0 ,qo) is the initial state, Ej x Eo is the input 
of the shield, Eo is the output of the shield, S 9 is the next-state function, and F 9 is the 
set of safe states, such that 6 9 ((u, t, v', q), ( 07 , <to), o’o’) = 

(S u (u, {a 0 ,a 0 ')),5 v '(v', ( 07 , a 0 )), 6 q (q, ( 07 , a 0 '))), 

and F 9 = {(it, t, v', q) £ G \ v' £ F vl V ((q G F q ) A (it = ( w, 0) —► £ = f 0 ))}. 

In the definition of F®, the term v' f F v> reflects our assumption that V \= ip v . If 
this assumption is violated, then v' qL F v ' will hold forever, and our shield is allowed to 
behave arbitrarily. This is exploited by our synthesis algorithm to find a more succinct 
shield by treating such states as don’t cares. If v 1 G F v> , we require that q G F q , i.e., 
it is a safe state in Q, which ensures that the shield output will satisfy <p. The last term 
ensures that the shield can only deviate in the fc-step recovery period, i.e., while c ^ 0 
in U. If the design makes a second mistake within this period, U enters ue and arbitrary 
deviations are allowed. Yet, the shield will still enforce ip in this mode (unless T> [F p v ). 
Step 5. Solving the Safety Game: We use standard algorithms for safety games (cf. 
e.g. fT5l ) to compute a winning strategy p for Q. Then, we implement this strategy in a 
new reactive system S = (G, go, E, Eo, 5, p) with S(g, a) = S 9 (g, a, p(g , a)). S is the 
^■-stabilizing generic shield. If no winning strategy exists, we increase k and try again. 
In our experiments, we start with /,: = 1 and then increase k by 1 at a time. 

Theorem 1. Let ip = {<p \,..., ipi} be a set of critical safety properties pq = {Qi, < 70 ^, 
E, Si, Fj), and let ip v C ip be a subset of valid properties. Let \V\ = ]~[ \Qi\ be 
the cardinality of the product of the state spaces of all properties of ip v . Similarly, let 
1^1 = \Qi\. A k-stabilizing generic shield with respect to tp\ip v and <p v can 

be synthesized in 0(k 2 • 2 2 ^! • |y| 4 • |f?| 2 ) time (if one exists). 

Proof. Safety games can be solved in 0(x + y) time fl5l . where x is the number of 
states and y is the number of edges in the game graph. Our safety game Q has at most 
x = ((k + 1) • + 1) • (2 ■ |y|) • 2 • (\R\ ■ |I/|) states, so at most y = x 1 edges. □ 

Variations. The assumption that no second violation occurs within the recovery period 
increases the chances that a /^-stabilizing shield exists. However, it can also be dropped 
with a slight modification of U in Step 1: if a violation is committed and c > 1, we set c 
to k instead of visiting ue■ This ensures that synthesized shields will handle violations 
within a recovery period normally. The assumption that the design meant to give one of 
the allowed outputs if a violation occurs can also be relaxed. Instead of continuing to 
monitor the behavior from the allowed next states, we can just continue from the set of 
all states, i.e., traverse to state ( R, k ) in U. The assumption that V \= <p v , i.e., the design 
satisfies some properties, is also optional. By removing V and V', the construction can 
be simplified at the cost of less implementation freedom for the shield. 

By solving a Biichi game (which is potentially more expensive) instead of a safety 
game, we can also eliminate the need to increase k iteratively until a solution is found. 
This is outlined in Appendix [A] 
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6 Experiments 


We have implemented the fc-stabilizing shield synthesis procedure in a proof-of-concept 
tool. Our tool takes as input a set of safety properties, defined as automata in a simple 
textual representation. The product of these automata, as well as the subset construction 
in Step 1 of our procedure is done on an explicit representation. The remaining steps 
are performed symbolically using Binary Decision Diagrams (BDDs). Synthesis starts 
with k = 1 and increments k in case of unrealizability until a user-defined bound is 
hit. Our tool is written in Python and uses CUDD JT] as the BDD library. Our tool can 
output shields in Verilog and SMV. It can also use the model checker VIS (6) to verify 
that the synthesized shield is correct. 

We have conducted three sets of experiments, where the benchmarks are (1) selected 
properties for a traffic light controller from the VIS J6| manual, (2) selected properties 
for an ARM AMBA bus arbiter j4), and (3) selected properties from LTL specifica¬ 
tion patterns OH. None of these examples makes use of ip v , i.e., tp v is always empty. 
The source code of our proof-of-concept synthesis tool as well as the input files and 
instructions to reproduce our experiments are available for downloacQ. 

Traffic Light Controller Example. We used the safety specification in Fig. [I] as input, 
for which our tool generated a 1-stabilizing 
shield within a fraction of a second. The shield 
has 6 latches and 95 (2-input) multiplexers, 
which is then reduced by ABC G) to 5 latches 
and 41 (2-input) AIG gates. However, most of 
the states are either unreachable or equivalent. 

The behavior of the shield is illustrated in Fig. [8] 

Edges are labeled with the inputs of the shield. Red dashed edges denote situations 
where the output of the shield is different from its inputs. The modified output is writ¬ 
ten after the arrow. For all non-dashed edges, the input is just copied to the output. 
Clearly, the states X, Y, and Z correspond to H, B, and F in Fig.0 

We also tested the synthesized shield using the traffic light controller of lfj~6ll . which 
also appeared in the user manual of VIS J6). This controller has one input (car) from 
a car sensor on the farm road, and uses a timer to control the length of the different 
phases. We set the “short” timer period to one tick and the “long” period to two ticks. 

The resulting behavior with¬ 
out preemption is visualized in 
Rg. m where nodes are labeled 
with names and outputs, and 
edges are labeled with conditions 
on the inputs. The red dashed ar¬ 
row represents a subtle bug we 
introduced: if the last car on the 
farm road exits the crossing at a rare point in time, then the controller switches from rg 
to gr without passing rr. This bug only shows up in very special situations, so it can go 
unnoticed easily. Preemption is implemented by modifying both directions to r without 

2 

http://www.iaik.tugraz.at/content/research/design_verification/others/ 



Fig. 9: Traffic light implementation. 



Fig. 8: Traffic light shield. 
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Fig. 10: Guarantee 3 from 0. Fig. 11: Shield execution results. 


changing the state if p = 1. We introduced another bug here as well: only the highway 
is switched to r if p = 1, whereas the farm road is not. This bug can easily go unnoticed 
as well, because the farm road is mostly red anyway. The following trace illustrates how 
the synthesized shield handles these errors: 
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The first bug strikes at Step 7. The shield corrects it with output IT. A 2-stabilizing shield 
could also have chosen rg, but this would have made a second deviation necessary in 
the next step. Our shield is 1-stabilizing, i.e., it deviates only at the step of the violation. 
After this correction, the shield continues monitoring the design from both state F and 
state B of Fig. 0 as explained earlier, to detect future errors. Yet, this uncertainty is 
resolved in the next step. The second bug in Step 12 is simpler: outputting IT is the only 
way to correct it, and the next state in Fig. 0 must be B. 

When only considering the properties 1 and 2 from Section0 the synthesized shield 
has no latches and three AIG gates after optimization with ABC f7| . 

ARM AMBA Bus Arbiter Example. We used properties of an ARM AMBA bus ar¬ 
biter 0| as input to our shield synthesis tool. Due to page limit, we only present the 
result on one example property, and then present the performance results for other prop¬ 
erties. The property that we enforced was Guarantee 3 from the specification of 0, 
which says that if a length-four locked burst access starts, no other access can start un¬ 
til the end of this burst. The safety automaton is shown in Fig. [10] where B, s and R 
are short for hmastlock A HBURST=BURST4, start, and HREADY, respectively. 
Lower case signal names are outputs, and upper-cases are inputs of the arbiter. S,, : is 
unsafe. SO is the idle state waiting for a burst to start (B A s). The burst is over if input 
R has been true 4 times. State Si, where i = 1,2,3,4, means that R must be true for i 
more times. The counting includes the time step where the burst starts, i.e., where SO is 
left. Outside of SO, s is required to be false. 

Our tool generated a 1-stabilizing shield within a fraction of a second. The shield 
has 8 latches and 142 (2-input) multiplexers, which is then reduced by ABC 0 to 4 
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latches and 77 AIG gates. We verified it against an arbiter implementation for 2 bus 
masters, where we introduced the following bug: the design does not check R when the 
burst starts, but behaves as if R was true. This corresponds to removing the transition 
from SO to S4 in Fig.|T0] and going to S3 instead. An execution trace is shown in Fig.lUI 
The first burst starts with s = true in Step 3. R is false, so the design counts wrongly. 
The erroneous output shows up in Step 7, where the design starts the next burst, which 
is forbidden, and thus blocked by the shield. The design now thinks that it has started 
a burst, so it keeps s = false until R is true 4 times. Actually, this burst start has been 
blocked by the shield, so the shield waits in SO. Only after the suppressed burst is over, 
the components are in sync again, and the next burst can start normally. 

To evaluate the performance of our tool, 
we ran a stress test with increasingly larger 
sets of safety properties for the ARM AMBA 
bus arbiter in (4). Table[l]summarizes the re¬ 
sults. The columns list the number of states, 
inputs, and outputs, the minimum k for 
which a ^-stabilizing shield exists, and the 
synthesis time in seconds. All experiments 
were performed on a machine with an Intel 
i5-3320M CPU@2.6 GHz, 8 GB RAM, and 
a 64-bit Linux. Time-outs (G2+3+4, Gl+2+4+5 and Gl+3+4+5) occurred only when 
the number of states and input/output signals grew large. However, this should not be 
a concern in practice because the set of critical properties of a system is usually much 
smaller, e.g., often consisting of invariance properties with a single state. 

LTL Specification Patterns. 

Dwyer et al. m studied the 
frequently used LTL speci¬ 
fication patterns in verifica¬ 
tion. As an exercise, we ap¬ 
plied our tool to the first 10 
properties from their list J2) 
and summarized the results in 
Table [2] For a property con¬ 
taining liveness aspects (e.g., 
something must happen even¬ 
tually), we imposed a bound 
on the reaction time to obtain 
the safety (bounded-liveness) 
property. The bound on the 
reaction time is shown in Col¬ 
umn 3. The last four columns 
list the number of states in the 
safety specification, the synthesis time in seconds, and the shield size (latches and AIG 
gates). Overall, our method runs sufficiently fast on all properties and the resulting 
shield size is small. We also investigated how the synthesis time increased with an in- 


Table 2: Synthesis results for the LTL patterns 
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Table 1: Performance for AMBA (4J. 
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creasingly larger bound b. For Property 8 and Property 6, the run time and shield size 
remained small even for large automata. For Property 10, the run time and shield size 
grew faster, indicating room for further improvement. As a proof-of-concept imple¬ 
mentation, our tool has not yet been optimized specifically for speed or shield size - we 
leave such optimizations for future work. 


7 Conclusions 

We have formally defined the shield synthesis problem for reactive systems and pre¬ 
sented a general framework for solving the problem. We have also implemented a new 
synthesis procedure that solves a concrete instance of this problem, namely the synthe¬ 
sis of fc-stabilizing generic shields. We have evaluated our new method on two hardware 
benchmarks and a set of LTL specification patterns. We believe that our work points to 
an exciting new direction for applying synthesis, because the set of critical properties 
of a complex system tends to be small and relatively easy to specify, thereby making 
shield synthesis scalable and usable. Many interesting extensions and variants remain 
to be explored, both theoretically and experimentally, in the future. 
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A Synthesis of Stabilizing Generic Shields 


In this section, we present a method for synthesizing fc-stabilizing shields with arbitrary 
but finite fc. We call such shields stabilizing (without the “k”). A synthesis procedure for 
stabilizing shields is also useful as a preprocessing step if we want to enforce a particu¬ 
lar (or minimal) k: Even for a realizable specification, the /.'-stabilizing shield synthesis 
problem may be unrealizable for any finite k. When specification ip is realizable, there 
exists a reactive system V such that V |= <p. However, it does not mean that a shield 
S exists for any design V, such that (Vo S) \= and (Vo S) deviates from V for at 
most k time steps. 

Example 3. Consider the safety specification on the right, where o\ and 02 are 
outputs, and r x is unsafe. The design must pro¬ 
duce either 01 A -102 globally or ~<Oi globally. 

The fc-stabilizing shield synthesis problem is 
unrealizable for any finite k: if the design pro¬ 
duces o\ A 02 initially, the shield must deviate to 
either <> \ A -102 or -> 02 . In the former case, the 
design could produce -1 o 1 from that point on, in the latter case <> \ A - 1 O 2 . This would 
cause an indefinite deviation with only a single violation. □ 

Whether a /^-stabilizing shield exists for some finite k is difficult to detect with the 
synthesis procedure from Section [T2l In case of unrealizability of the shield for a given 
fc, we cannot know if we just need to increase k, or if no finite k would work. The 
synthesis process presented in the following sub-section will decide the realizability 
problem. We can also synthesize a stabilizing shield, measure its k, and minimize this 
k further with the procedure from Section l572l until we hit the unrealizability barrier. 

A.l Construction for Synthesizing Stabilizing Shields 

A generic stabilizing shield can be synthesized (if one exists) with only a few modifi¬ 
cations to the procedure from Section [b~2l Instead of a counter c € {0,..., fc}, we use 
a counter d e {0,1, 2} with only three different values. Intuitively, d = 2 is an abstrac¬ 
tion for c > 1. We construct a Biichi game that is won if d < 1 infinitely often (and all 
the other shield requirements are satisfied). A Biichi game is like a safety game, but the 
given set of final states must be visited infinitely often for the system to win the game. 
A winning strategy for this Biichi game corresponds to a /^-stabilizing shield with some 
finite k, and the k can even be computed during synthesis. The construction is similar 
to Section [T2l with only a few modifications: 

Step 1. Instead of using a counter c € {0,..., /.:}, we use a three-valued counter d € 
{0,1,2} to track whether we are currently in the recovery phase or not. Intuitively, 
d = 2 if c would be > 1. That is, d is 0 initially. If d < 2 and the design makes a mistake 
(leaves W r ), then d is set to 2. If it was already 2, we enter up:- In order to decide when 
to decrement d from 2 to 1, we add a special output r to the shield. If this output is set to 
true and d = 2, then d is set to 1 in the next step. The behavior for d = 1 is the same as 
in Section [A2| if another violation occurs, d is set to 2. Otherwise, d is decremented to 
1. We denote this slightly modified violation monitor by U' = ((/', u' 0 , £ x 2^ ,5 U ') 
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with U' = ( 2 r x {0,1,2 })Uu £ . The subsequent steps will ensure that the shield will 
only be allowed to deviate if d > 0 in the next step. We will also require that d cannot 
be 2 indefinitely. 

Step 2 and Step 3 are performed as described in Section I5T21 

Step 4. We construct a Biichi game Q' = (G", g ' 0 , Si x So , So x 2^ r \ S 9 ', F 9 ') as the 
synchronous product of U ', T, V and Q as follows: 

- G' = U' x T x V' x Q x 1 x B, 

-30 = (u' 0 , to, v' 0 ,q 0 , false, false). 

- 6 g '((u , ,t,v',q,m,n),(ai,a 0 ),(a 0 , ,r )) = (S u '(u', ((07, <j 0 ), r)), 

S*(t, ( ao,cr 0 ')),S v '(v(ai,a 0 )),S 9 (q, (ai,cj 0 ')),m',n'), where 

• m! = true iff to = true or q ^ F 9 

• n’ = true iff n = true or u' = (w, 0) A t = t%, and 

- F 9 ' = {(V, t, vq, m, n) £ G' \ v' qL F v> V ( ~>n A ~>m A d < 1)}. 

The intuition behind this construction is as follows. We extend the state space of the 
synchronous product by two bits, m and n. The bit to is true if the execution has ever 
visited an unsafe state in Q. The bit n is true if there has been an illegal deviatiorQ. 
With this information, the accepting states (that need to be visited infinitely often) are 
then defined as follows. Outside of F v ', everything is accepting. This makes sure that 
the shield can behave arbitrarily if V ip v . Otherwise, a state is accepting if d < 1 
(the last recovery period is over), to is false CD o S \= <p so far) and n is false (no illegal 
deviations so far). Visiting F 9 ' infinitely often implies that recovery periods are over 
infinitely often, and to and n are never true (these bits cannot change back to false). 
Step 5. Just like safety games, Biichi games also have a memoryless strategy. We com¬ 
pute such a strategy and implement it as described in Section 15.21 If no such strategy 
exists (which is easy to detect during synthesis), then this is reported to the user. 
Discussion. Note that the Biichi objective ensures that recovery phases are over in¬ 
finitely often, but not that they are bounded in time. There may exist a strategy to sat¬ 
isfy the Biichi objective without any finite bound on the recovery time. E.g., the first 
recovery phase could take 2 steps, the second one 4 steps, the third one 8 steps, etc. 
However, such a strategy would require infinite memory. We construct and implement 
a memoryless strategy, which guarantees a bounded recovery. We can even measure the 
maximum length of any recovery phase while synthesizing the shield: Biichi games can 
be solved with a doubly-nested fixpoint computation ED. The number of iterations of 
the inner fixpoint (in the last iteration of the outer fixpoint) corresponds to the maxi¬ 
mum number of steps needed to reach a state of F 9 ', i.e., a state where the recovery is 
over. Hence, this value is also the maximum length of a recovery period, i.e., the value 
k for the resulting fc-stabilizing shield. 


3 Recall that u' = ( w, 0) means that the counter d introduced in Step 1 is 0, i.e., no deviation 
was allowed in the previous time step; t = ti indicates that a deviation has occurred in the 
previous time step. 
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